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@ System and method for synchronization of multimedia streams. 



@ A computer-based multimedia presentation sys- 
tem is provided with a synchronization scheme for 
recording and playing independent media. The dis- 
closed system and nhethod allows media processes 
and single medium processes to achieve and main- 
tain synchronization with each other without process 
interdependence and without interprocess commu- 
nication. This capability is provided by assigning a 
common clock for all processes, requiring all partici- 
pating media processes to reference the common 



clock, informing each process of a synchronization 
basepoint called a "zero-time", and then allowing 
each process to independently synchronize itself to 
the common clock. The common clock itself does 
not provide any stimulus to a media process; it is a 
passive component in the synchronization. The me- 
dia process is the active component, referring to the 
common clock as required to maintain synchroniza- 
tion for the particular media it is handling. 
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This invention relates In general to multiproces- 
sing computer systems and in particular to syn- 
chronization of multiple independent processes in 
such computer systems. 

The dramatic advances being made in com- 
puter processing speed and storage capacity have 
created new opportunities for computer applica- 
tions. One of these opportunities lies in the area of 
multimedia presentations, which combine comput- 
erized audio, still pictures, video, text, and other 
media to create dynamic, sensory-rich communica- 
tion with computer users. 

One of the important challenges involved in 
presenting multimedia information in a cohesive, 
life-like fashion is synchronization of the numerous 
components that make up a typical presentation. 
This synchronization includes two aspects: first, the 
presentation components must start in synchrony; 
second, they must progress at the same rate. 
Moreover, both aspects of synchronization must be 
adhered to even when the presentation is played 
after fast-forward or reverse, or is started at a point 
other than its beginning. 

Because the modem computer systems used 
to play multimedia presentations frequently run 
multiple tasks, programs, and processes concur- 
rentiy, and use multiple processors simultaneously, 
synchronization of processes can become a signifi- 
cant problem. Thus, the audio portion of a pre- 
sentation, which comprises one process, may pro- 
ceed faster than the video portion, which comprises 
a second process, for reasons which cannot be 
known prior to playing the presentation. For exam- 
ple, if a multimedia presentation which plays the 
audio and video of a human speaker is interrupted 
by an extraneous concurrent process, the audio 
and video can become unsynchronized, resulting in 
the perception by the observer that the speaker's 
lip movements and words do not match. Such 
effects can significantly detract from the overall 
quality of the presentation. 

Various techniques have been devised for deal- 
ing with the synchronization problems that arise In 
multimedia presentation systems. There are three 
methods which have been user to solve the syn- 
chronization problem. The first of these is a barrier 
method, that is. each media is allowed to proceed 
at its own pace until a barrier time or "sync-point" 
is encountered. Each media waits at the sync-point 
until all other media have amVed. Once all media 
have reached tiie sync-point the media are again 
allowed to proceed at there own pace. 

A second approach uses messages or 
"pulses" to indicate to the component media, the 
current position, time, or offset in the multimedia 
presentation. A master process is sends out these 
pulses at some rate. Each medium is responsible 
for making adjustment to its rate to try to match the 



current or anticipated position (time) of the master. 

The third approach is to use a common clock 
which is adjustable by a master process (or pro- 
cesses). Although such methods have been openly 

5 discussed, we are unaware of any complete al- 
gorithm which includes methods for initiating, and 
re-initiating, the synchronization and allows for "re- 
winding" and "fast-forwarding" in the presentation. 
While these various approaches all tend to 

10 improve the synchronization of multimedia presen- 
tations vis-a-vis a free-running paradigm, they are 
not without serious drawbacks. For example, the 
sync-pointing approaches can be dependent on the 
processor speed of the hardware. This may allow a 

75 presentation to work well in some cases but fail to 
synchronize in other cases or on different hard- 
ware. The pulsing approaches require the pulses to 
be sent out at a sufficient rate to allow for synchro- 
nization (this rate may vary on the presentation) 

20 video for example may require significant pulse, 
requiring signiticant system resources. Pulsing also 
requires that processes quickly identify the recep- 
tion of a pulse. Unfortunately, pulse may arrive at 
times which are inconvenient for the receiving pro- 

25 cess. Other common-clock approaches may not 
have clear startup or re-start methods. AH of the 
methods described above have some sort of mas- 
ter process, the rate of that process is assumed to 
be correct. However, even a master process run- 

30 ning at a high priority will experience some vari- 
ance in its rate. This variance make the task of the 
slave processes even more difficult, not only do 
they have to adjust for there own variance, they 
must also adjust to the variance in the master 

35 process's execution rate. The master process also 
becomes a single point of failure. The methods 
described above all rely on a clock with encoded 
clock values. These clock values are typically 
stored in fixed length fields. With out some care. 

40 these clock values may be subject to wrapping, 
just as a regular clock wraps from 12:00 back to 
1 :00. Also, the methods described above, may not 
be applicable to the recording of a synchronized 
presentation. 

45 Thus, there has heretofore existed an unmet 

need for a system and method of synchronizing 
multiple processes in a multimedia presentation 
without the use of an active clock or a resident 
controlling process. 

50 In accordance with the present invention there 

Is now provided a method for synchronizing initi- 
ation of media operations in a multimedia recording 
and playback system, comprising the steps of: in a 
starter process, receiving the current time from a 

55 clock means, assigning the received current time 
as an initialization-time value, determining a zero- 
time value by adding a process preparation time to 
the initialization-time value, broadcasting the zero- 
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time value to at least one media process; and in 
the media process, receiving the zero-time value, 
upon the current time reaching the zero-time value, 
initiating the media operation. 

Viewing the present invention from a second 
aspect, there is now provided a method for syn- 
chronizing initiation of media operations in a mul- 
timedia recording and playback system, comprising 
the steps of: in a starter process, receiving the 
current time from a clock means, assigning the 
received current time as an initialization-time value, 
determining a zero-time value by adding a process 
preparation time to the initialization-time value, 
broadcasting the zero-time value to at least one 
media process; and in the media process, receiv- 
ing the zero-time value, determining a media start- 
time by combining an offset time with the zero-time 
value, and upon the current time reaching the me- 
dia start-time, initiating the media operation. Prefer- 
ably, the media start-time is determined by adding 
the offset time to the zero-time value. 

Viewing the present invention from a third as- 
pect, there is now provided a method for synchro- 
nizing media recording in a multimedia recording 
and playback system, comprising the steps of: in a 
starter process, receiving the current time from a 
clock means, assigning the received current time 
as an initialization-time value, determining a zero- 
time value by adding a process preparation time to 
the initialization-time value, broadcasting the zero- 
time value to at least one media process; and in 
the media process, receiving the zero-time value, 
determining a media start-time by adding an offset 
time to the zero-time value, upon the current time 
reaching the media start-time, repeating for each 
media event the sub-steps of obtaining the current 
time from the clock means, determining an event 
offset based on the zero-time value and the current 
time, logging the event offset in association with 
the media event, recording the media event. 

Viewing the present invention from a fourth 
aspect, there is now provided a method for syn- 
chronizing media playback in a multimedia record- 
ing and playback system, comprising the steps of: 
in a starter process, receiving the current time from 
a clock means, assigning the received current time 
as an initialization-time value, determining a zero- 
time value by adding a process preparation time to 
the initialization-time value; broadcasting the zero- 
time value to at least one media process; and in 
the media process, receiving the zero-time value, 
accessing a log of event offset times associated 
with a plurality of media events, for each logged 
event offset time, performing the sub-steps of 
readying the associated media event for playing, 
and when the event offset time arrives, playing the 
media event. 



Viewing the present invention from a fifth as- 
pect, there is now provided a method for synchro- 
nizing initiation of events in a multiprogramming 
computer system, comprising the steps of: In a 

5 starter process, receiving the current time from a 
clock means, assigning the received current time 
as an initialization-time value, determining a zero- 
time value by adding a process preparation time to 
the initialization-time value, broadcasting the zero- 

10 time value to at least one receiving process; and in 
the receiving process, receiving the zero-time val- 
ue, upon the current time reaching the zero-time 
value, initiating the event. 

Viewing the present invention from a sixth as- 

15 pect. there is now provided a method for synchro- 
nizing events in a multiprogramming computer sys- 
tem, comprising the steps of: in a starter process, 
receiving the current time from a clock means, 
assigning the received current time as an initializa- 

20 ti on-time value, determining a zero- time value by 
adding a process preparation time to the initializa- 
tion-time value; broadcasting the zero-time value to 
at least one receiving process; and in the receiving 
process, receiving the zero-time value, upon the 

25 current time reaching the zero-time value, repeat- 
ing for each event the sub-steps of executing the 
event, determining a next-event time, waiting until 
the current time reaches the next-event time. 

In a preferred embodiment of the present in- 

30 vention, a computer-based multimedia presentation 
system is provided with a synchronization scheme 
which allows media processes and single medium 
processes to achieve and maintain synchronization 
with each other without process interdependence 

35 and without interprocess communication. This ca- 
pability is provided by assigning a common clock 
for all processes, requiring all participating media 
processes to reference the common clock, Inform- 
ing each process of a synchronization basepoint 

40 called a "zero-time", and then allowing each pro- 
cess to independently synchronize itself to the 
common clock. 

The common clock is used by all media pro- 
cesses in the computer system to control the rate 

45 at which the media is presented. The common 
clock is made accessible to all processes so that 
all can maintain synchronization, ft is referenced as 
needed by the various processes participating in a 
particular presentation and its value is used to 

50 determine when to present the next media event. A 
media event can be broadly viewed as that which 
stimulates human perception. The duration of a 
media event is determined by the Individual media 
processes. 

55 The coded zero-time given to each media pro- 

cess consists of a field the same size as the 
common clock's field. The zero-time field contains 
a binary pattem which is to be logically added to 
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the common clock. The result of the addition yields 
another time scale called the zero-time scale. 
When the common clock advances to zero-time the 
resulting zero-time scale value equals zero, be- 
cause ot the wrapping of the logical addition, the 
multimedia presentation begins. In this way, the 
system is protected against improper operation 
caused by wrapping of the common clock. 

The value of the common clock in conjunction 
with the coded zero-time is the basis by which a 
media process determines when to present the 
next media event. The common clock itself does 
not provide any stimulus to a media process; it is a 
passive component in the synchronization. The me- 
dia process is the active component. As the com- 
mon clock value increases, so does the value on 
the zero-time scale. The zero-time scale represents 
the time elapsed since the start of the presentation. 
A media process determines when to present the 
next media event by comparing the time the event 
needs to be presented with the current zero-time 
scale value. The process then waits until the time 
of the next event equals the value on the zero-time 
scale and at that time presents the event. 

Additional aspects of the invention include al- 
gorithms for starting, re-starting, rewinding, fast- 
forwarding, and recording presentations. 

Numerous advantageous effects flow from the 
zero-time synchronization technique of the present 
invention. First, media processes are able to work 
in concert with one another without the necessity 
for intercommunication. By synchronizing itself to 
the common clock, each process is automatically 
synchronized with all other processes which them- 
selves are synchronized with the common clock. 
Since each process has independent control of 
synchronization, each process can determine its 
own optimum times for checking synchronization. 
Second, because the coded zero-time establishes 
a point in time which each process must use as 
time zero (i.e. 00:00:00), a process can be started 
either before or after time zero and is still able to 
synchronize with the other media processes. More- 
over, processes can re-synchronize themselves by 
performing a resynchronization action such as 
sleeping, compressing, truncating, deleting, or 
elongating because they can detemnine the syn- 
chronization discrepancy by comparing the time 
within the presentation to the time on the zero-time 
scale. Sleeping involves waiting for the next-event 
time. Compressing ts accomplished by performing 
the media at a higher rate than originally recorded. 
Truncating and deleting are methods of not per- 
forming certain parts of a multimedia presentation. 
Elongating is accomplished by performing the me- 
dia at a lower rate than originally recorded. Third, 
because only the common clock is required for 
synchronization, processes do not need operating 
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system support for synchronization or for inter- 
process communication generally. Fourth, since the 
coded zero-time augments the information in the 
common clock for each presentation, a single clock 
5 can t>e used for multiple concurrent presentations. 
Finally, because the coded information allows the 
common clock to appear as time zero, the entire 
range of the clock can be used before the clock 
wrcips to its initial value. Thus, long presentations 
70 can be created without concern for the timing con- 
fusion caused by clock wrapping. 

A preferred embodiment of the present Inven- 
tion will now be described with reference to the 
accompanying drawings in which: 
IS Figure 1 is a block diagram illustrating the major 
processes contributing to the synchronization of 
multimedia presentations in accordance with the 
present invention; 

Rgure 2 is a timeline illustrating the components 

20 used to develop zero-time; 

Figure 3 is a timeline illustrating the synchro- 
nized recording of multi-media; 
Figure 4 is a timeline illustrating the synchro- 
nized playback of multi-media; 

25 Figure 5 is a timeline illustrating the synchro- 
nized playback and recording of multi-media; 
Figure 6 is a timeline illustrating the determina- 
tion of zero-time and creation of coded zero- 
time; 

30 Figure 7 is a timelinei illustrating the use of 
coded zero-time; and 

Rgure 8 Is a timeline illustrating the use of zero- 
time in media repositioning operations. 
Shown in Fig. 1 is a high level representation 

35 of the system components and information flow 
which provide multimedia synchronization in accor- 
dance with the present invention. The major com- 
ponents include: starter process 100 which per- 
forms the functions of determining zero-time at 101 

40 and initiating the media processes at 103. media 
processes 104 105 106, also known as media 
operations, which record and/or playback a syn- 
chronized multimedia presentation, and clock 
means 102 which provides ttie current time in the 

45 computer system. The current time is the inter- 
pretation of the temporal bit pattern into standard 
time nomenclature hours, minutes, seconds, mil- 
liseconds, etc. 

The system executes on computer processor 

50 180. recording and playing media to/from computer 
storage 182. Processor 180 may be a microproces- 
sor or any other instruction-executing computer 
processor means. Computer storage 182 may be 
RAM memory, disk memory, optical, tape, bubble, 

55 or any other form of electronically accessible data 
storage. Normally, currently accessed information 
Including media is stored in RAM where it is read- 
ily available to processor 180. Other data, such as 

4 
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media not currently being presented or recorded is 
stored on disk or tape where it is still accessible 
albeit at a lower rate. 

In operation, starter process 100. executing on 
computer processor 180. first determines zero-time 
at 101 by gathering estimated initialization times 
108. 109, and 110 from participating media pro- 
cesses 104. 105, and 106. This can be accom- 
plished by a variety of techniques, including 
querying each process for an estimated initializa- 
tion time or encapsulating the estimated initializa- 
tion times within starter process 100. The esti- 
mated initialization time for the media processes is 
then calculated. Next, clock 102 is used to obtain a 
reference to the current time via 107. this is also 
known as the initialization-time value. By adding 
together the current time with the estimated initial- 
ization time for the media processes, known as a 
process preparation time, starter process 100 de- 
termines a zero-time value and constructs the cod- 
ed zero-time. The determined zero-time, now in the 
form of coded zero-time 111, is then used to ini- 
tiate the media processes at 103. This involves 
dispatching each of media processes 104, 105, and 
108. while passing to them coded zero-time via 
112. 113. and 114. Each of the media processes 
104. 105. and 106. acts as a receiving process in 
that they receive the coded zero-time 111. This 
completes communication required to create syn- 
chronization. Given this single receipt of zero-time, 
each media process can proceed to perform its 
function without further input from starter process 
100. To achieve and maintain synchronization, 
each of media processes 104, 105, and 106 uses 
the zero-time information in conjunction with re- 
peated references to the current time from clock 
102 via 115. 116, and 117. 

The basic components necessary for synchro- 
nizing multimedia processes in accordance with the 
present invention are shown in Fig. 2. Included are 
common clock 201, zero-time register 204, zero- 
time broadcast means 203. and the zero-time scale 
234. The common clock 201 is accessible to all 
participating processes. It advances at a predict- 
able rate, and has a frequency suitable for mul- 
timedia presentations. If a clock having a period 
(i.e. the clock can "wrap") is chosen for common 
clock 201. that period must be longer than the 
multimedia presentation in order to avoid the ap- 
parent anomaly that occurs when the time progres- 
sion experiences a step function from a high value 
to a low value. In the preferred embodiment, the 
standard system clock of an IBM PS/2 personal 
computer is used to generate the common clock. 
This clock has a period of over 49 days, which is of 
ample length in comparison with virtually all mul- 
timedia presentations. 
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Zero-time register 204 is loaded with data, de- 
termined in a manner that will be described in 
detail below. Identifying a time at which the mul- 
timedia presentation is to begin. This point in time 

5 will be referred to as zero-time 205. Zero-time 
broadcast means 203 is loaded with a value, deter- 
mined in a manner that will be described in detail 
below, identifying which is broadcast to the various 
participating media processes and from which each 

10 process can determine zero-time 205 and the zero- 
time scale 234. This data will be referred to as 
coded zero-time 202. 

Shown in Rg. 3 is a multimedia recording 
system in accordance with the present invention. 

75 Two major components are included in the system. 
The first is a process which establishes zero-time 
301 and creates the coded zero-time 302. This 
process is called the recording starter process, and 
is designated as 303. From the coded zero-time 

20 302 and the common clock 201 a zero-time scale 
334 can be constructed by any process. The sec- 
ond component is one or more processes which 
will use coded zero-time 302 in conjunction with 
common clock 201 in recording multimedia or sin- 

25 gle media events. These processes are called me- 
dia recording processes and are designated as 
304. Examples of multimedia recording processes 
304 are audio recording process 305, text record- 
ing process 306, and image recording process 307. 

30 During operation of the system, common clock 

201 advances continuously. Recording starter pro- 
cess 303 is started to establish zero-time 301 dur- 
ing zero-time initialization 308 and initiate all of 
media recording processes 304 during process ini- 

35 tiation 335. Recording starter process 303 calcu- 
lates zero-time 301 based upon the current time 
322 on common clock 201 and the estimated pro- 
cess preparation 309 needed by media recording 
processes 304. Recording starter process 303 con- 

40 structs coded zero-time 302 and initiates audio 
recording process 305, text recording process 306. 
and image recording process 307, passing in cod- 
ed zero-time 302 to each of media recording pro- 
cesses 304 Alternatively, recording starter process 

45 303 may keep coded zero-time 302 available for 
query by any of media recording processes 304. 
After all media recording processes 304 have fc>een 
initiated, recording starter process 303 can either 
cease, become a member of media recording pro- 

50 cesses 304. or perform other tasks. 

Each of media recording processes 304 will 
perform process preparation, designated as 310. in 
order to prepare Internal resources and acquire 
external resources. If zero-time 301 has not passed 

55 when one of media recording processes 304 has 
finished process preparation 310, that process will 
perform a process wait, designated as 311. until 
zero-time 301 occurs. This time is known as media 
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start-time or more generically a first event start- 
time. Each of media recording processes 304 will 
operate independently of the other media recording 
processes. This Is possible because each of media 
recording processes 304 knows of common clock 
201 and possesses coded zero-time 302. 

Once audio recording process 305 has been 
initiated, it begins audio recording process prepara- 
tion, designated as 312. Audio recording process 
305 will acquire control of the audio recording 
hardware, allocate filename(s), prepare a place to 
put the digitized data, and initialize internal control 
mechanisms. If zero-time 301 has not passed by 
the time audio recording process preparation 312 
is finished, audio recording process 305 will cal- 
culate the amount of time it must perform audio 
recording process wait, designated as 313. in order 
to begin the multimedia at zero-time 301. The 
recorded data from the audio recording process 
305 contains Information indicating that recording 
started at zero-time 301. If zero-time 301 has 
passed by the time audio recording process prep- 
aration 312 is finished, the recorded data from 
audio recording process 305 contains the knowl- 
edge that recording started after zero-time 301 by 
the difference in time between zero-time 301 and 
the end of audio recording process preparation 
312. In all cases the ACTUAL offset, not the in- 
tended offset from zero time is recorded for the 
event. This offset, the offset time, is used during 
play-back to synchronize the media just as they 
were observed during recording. 

Once text recording process 306 has been 
initiated, it begins text recording process prepara- 
tion, designated as 314. 

Likewise, image recording process 307 will begin 
image recording process preparation, designated 
as 315. at this time. As with audio recording pro- 
cess preparation 312, each process must allocate 
its own internal and external resources. Text re- 
cording process 306 must then determine whether 
to enter text recording process wait, designated as 
316; likewise, image recording process 307 must 
determine whether to enter image recording pro- 
cess wait, designated as 317. Each of media re- 
cording processes 304 keeps track of whether it 
started at zero-time 301 or how long after zero-time 
301 it started. 

As audio recording process 305. text recording 
process 306. and image recording process 307 are 
recording, they periodically check the difference 
between zero-time 301 and current time 318. This 
difference, denoted as relative time 319, is used to 
adjust the rate of recording, to log the time of an 
event, known as an event offset, or to determine 
the duration of an event. An alternative to logging 
the time of certain media events is keeping a 
constant rate at which the media events are re- 



corded. From the rate, information similar to a log 
can be constructed. 

Shown in Rg. 4 is a multimedia playing system 
in accordance with the present invention. Two ma- 

5 jor components are included in the system. The 
first is a process which establishes zero-time 401 
and creates coded zero-time 402. This process is 
called the playback starter process, and is des- 
ignated as 403. From the coded zero-time 302 and 

10 the common clock 201 a zero-time scale 434 can 
be constructed by any process. The second com- 
ponent is one or more processes which use coded 
zero-time 402 in conjunction with common clock 
201 to play multimedia or single media events. 

75 These processes are called media playt>ack pro- 
cesses and are designated as 404. Examples of 
media playback processes 404 are audio playback 
process 405. text playback process 406. and image 
playback process 407. 

20 As common clock 201 is advancing, playback 

starter process 403 Is started to establish zero-time 
401 during zero-time initialization 408. and initiate 
media playback processes 404 during process initi- 
ation 435. Playback starter process 403 calculates 

25 zero-time 401 based upon the current time 422 on 
common clock 201 and the estimated process 
preparation 409 needed by media playback pro- 
cesses 404. Playback starter process 403 con- 
structs coded zero-time 402 and initiates audio 

30 playback process 405, text playback process 406, 
and image playback process 407. passing in coded 
zero-time 402 to each of media playback pro- 
cesses 404. Alternatively, playback starter process 
403 may retain coded zero-time 402 available for 

35 query by any of media playback processes 404. 
After media playback processes 404 have been 
initiated, playback starter process 403 can either 
cease, become a member of media playback pro- 
cesses 404, or perform other tasks. 

40 Each of media playback processes 404 will 

perform a process preparation 410 in order to 
prepare all of its internal resources and acquire alt 
of its external resources. If zero-time 401 has not 
passed when one of media playback processes 

45 404 has completed process preparation 410, that 
media playback process will perform a process 
wait 411 until zero-time 401 occurs. This time is 
known as media start-time. Each of media playback 
processes 404 will operate independentiy of the 

50 other media playback processes. This is possible 
because each of media playback processes 404 is 
aware of common clock 201 and is provided with 
coded zero-time 402. 

Once audio playback process 405 has been 

55 initiated, it begins audio playback process prepara- 
tion 412. Audio playback process 405 acquires 
control of the audio playback hardware, locates the 
audio data file(s) (including a log file in which the 
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28ro-time offsets are given for all audio events, 
such as. when to start the audio, and when to 
switch audio data files), prepares storage space for 
the digitized data, and initializes the audio playback 
hardware, any data buffers required for audio 
playback, as well as locating the common clock, 
determining the first audio event and any other 
general initialization. If zero-time 401 has not 
passed by the time audio playback process prep)- 
aration 412 is finished, audio playback process 405 
determines the amount of time it must perform 
audio playback process wait 413 in order to begin 
the multimedia presentation at zero-time 401. If 
zero-time 401 has passed by the time audio 
playback process preparation 412 is finished, audio 
playback process 405 advances into the recorded 
audio data to account for the difference in time 
between zero-time 401 and the end of audio 
playback process preparation 412. If the audio data 
contains information indicating that the original re- 
cording began after zero-time, audio playback pro- 
cess 405 continues its audio playback process wait 
413 until that amount of time after zero-time 401. 

Once text playback process 406 has been ini- 
tiated, it begins text playback process preparation 
414. Likewise, image playback process 407 begins 
image playback process preparation 415. As with 
audio playback process preparation 412, each pro- 
cess must allocate its internal and external re- 
sources. Text playback process 406 then deter- 
mines whether to enter text playback process wait 
416 until zero-time 401, whether to enter it until 
after zero-time 401 , or whether to advance through 
the text data. Image playback process 407 deter- 
mines whether to enter image playback process 
wait 417 until zero-time 401, whether to enter it 
until after zero-time 401, or whether to advance 
through the image data. 

As audio playback process 405, text playback 
process 406, and image playback process 407 are 
playing, they periodically check the difference be- 
tween zero-time 401 and current time 418. This 
difference, relative time 419. is used to adjust the 
rate of playing, and to detect the need to skip 
forward or wait for the next event in the case where 
synchronization is lost. In cases where an un- 
foreseen system load has caused the process to 
fall out of synchronization, the process must adjust 
the rate at which is plays to get back into sync. 
The process may choose to skip ahead if it has 
fallen behind in the presentation, or it may choose 
to wait if it has gotten ahead. Both skipping and 
waiting may be implemented using a variety of 
techniques known in the art. Waiting, for instance, 
may be implemented by polling or sleeping. Sleep- 
ing is chosen in the preferred embodiment be- 
cause it allows other processes to have the great- 
est possible access to computer resources. While 
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Skipping/waiting approach may be appropriate for 
an Image or video process, which can simply skip 
a frame, it is not the best choice for an audio 
process. Skipping forward in audio will result in a 

5 discontinuity in the audio. The preferred approach 
for audio is to increase the rate at which the audio 
plays. If the adjustment is small this temporary 
increase in the rate may not be noticeable. Simi- 
larly, the audio process may need to decrease the 

10 rate if the audio process has somehow become 
ahead of time. If large adjustments are needed In 
the audio rate, the frequency spectrum of the audio 
may be adjusted to preserve the tonal properties of 
the audio, using methods well known in the field of 

76 signal processing. 

Shown in Fig. 5 is a set of time lines depicting 
the sequence of adding multimedia to a presenta- 
tion. Three processes participate In this sequence. 
The first process establishes zero time at 501 and 

20 creates coded zero-time at 502. This process is 
designated as playback/record starter process 503. 
From the coded zero-time 302 and the common 
clock 201 a zero-time scale 534 can be construct- 
ed by any process. The second process is in 

25 practice one or more processes which use coded 
zero-time 502 in conjunction with common clock 
201 in playing multimedia or single media events. 
These processes are called media playback pro- 
cesses, designated generally as 504. Examples of 

30 media playback processes 504 include audio 
playback process 505, text playback process 506, 
and image playback process 507. The third pro- 
cess is also actually one or more processes, which 
in this instance use coded zero-time 502 in con- 

35 junction with common clock 201 in recording mul- 
timedia or single media events. These processes 
are called media recording processes 520. An ex- 
ample of a media recording process is image re- 
cording process 521 . 

40 As common clock 201 advances, 

playback/record starter process 503 is started to 
establish zero-time 501 during zero-time initializa- 
tion 508 and initiate media playback processes 504 
and media recording processes 520 during process 

45 initiation 535. Playback/record starter process 503 
calculates zero-time 501 based upon the current 
time 522 on common clock 201 and the estimated 
process preparation 509 needed by media 
playback processes 504 and media recording pro- 

50 cesses 520. Playback/record starter process 503 
constructs coded zero-time 502 and initiates audio 
playback process 505, text playback process 506. 
image playback process 507, and image recording 
process 521 , passing coded zero-time 502 to each 

55 of media playback processes 504 and media re- 
cording processes 520. Alternatively, playback/ 
record starter process 503 may keep coded zero- 
time 502 available for query by any of media 

7 
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playback processes 504 or media recording pro- 
cesses 520. After media playback processes 504 
and media recording processes 520 have been 
initiated, playback/record starter prcK:ess 503 can 
either cease, become a member of media playback 
processes 504, become a member of media re- 
cording processes 520, or perform other tasks. 

During this time, media playback processes 
504 and media recording processes 520 continue 
to function as described at)ove. The result of the 
simultaneous playback and recording of multimedia 
is that the newly recorded media will be in syn- 
chronization with the previously recorded media. 
Each of the recorded media has time synchroniza- 
tion information which can be used by a mul- 
timedia player having the ability to recognize com- 
mon clock 201 and coded zero-time 502. 

Shown in Fig. 6 is a set of timelines depicting 
the creation of coded zero-time. The process in- 
volves two principal components. The first is the 
current time on common clock 201 . This is denoted 
as current time 622. The second is information 
indicative of the anticipated startup times of media 
recording processes 620 and/or media playback 
processes 604. This information is used to con- 
struct estimated process preparation time 609. 

The calculation of zero-time is most easily un- 
derstood in the context of a specific example. For 
this purpose, reference will be made to the numeri- 
cal quantities in Rg. 6. It is to be noted that these 
quantities and the resulting calculations are pre- 
sented for illustrative purposes only. As contem- 
plated by the present invention, any clock base 
and timing may be selected. With reference to Fig. 
6, calculation of zero-time 601 begins with selec- 
tion of a common clock. The exemplary clock 
selected is 32-bits in length and has an increment 
representing one millisecond. Current time 622 on 
common clock 201 is 9:47:23.150, the equivalent of 
35,243,150 milliseconds. Estimated process prep- 
aration time 609 is 5 seconds, which is 5000 mil- 
liseconds. Zero-time 601 is calculated as current 
time 622 added to estimated process preparation 
time 609. The result is 35.248,150 milliseconds, or 
0219D816 in hexadecimal notation. 

Using a clock value of zero-time as the basis 
for a multimedia presentation has an inherent prob- 
lem, clock wrap. If the highest value of the clock is 
999:99:99.999 and zero-time is 999:98:00.000. the 
multimedia presentation must detect the wrapping 
of the clock and handle the exception in the next 
minute. By coding zero-time, the full range of the 
clock can be used without having to special case 
for pre-mature clock wrap. 

To create coded zero-time 602. also known as 
coding the zero-time value, the two's complement 
of zero-time 601 is calculated. Zero-time 601 is 
0219D816 (hex). The one's complement of zero- 



time 601 is FDE627E9 (hex). This quantity can be 
generated by XORing zero-time 601 with 
FFFFFFFF (hex). Next, the two's complement of 
zero-time 601 is FDE627EA (hex). This is deter- 

5 mined by adding 1 to the one's complement. Thus, 
coded zero-time 602 for this example is FDE627EA 
(hex). From the coded zero-time 602 and the com- 
mon clock 201 a zero-time scale 634 can be con- 
structed by any process. 

10 Shown in Fig. 7 is a timeline depicting the use 

of coded zero-time. The process involves four prin- 
cipal components. The first is coded zero-time 702, 
which is given to or obtained by process 723. From 
the coded zero-time 702 and the common clock 

75 201 a zero-time scale 734 can be constructed by 
any process. The second is the current time on 
common clock 201 . This is denoted as current time 
724- The third is a schedule of when each media 
event occurs, this is a list of event offsets and is 

20 designated as event schedule 725. This component 
could be a rate at which to perform media events 
rather than a list of discrete media events. The 
fourth component is a stream of data and a given 
rate at which to play the data. 

25 The general equation when using zero-time is: 

E-(T-Z) = C 

where E is the offset of the actual event within the 
30 presentation 726 729. T is the cun-ent time 724 730 
on the clock 201, Z is the zero-time 701 for the 
presentation, and C is the correction 732 733 need- 
ed by the media process 723 in the presentation of 
the media event to reproduce the timing of the 
35 actual event. In the case of playback, E, T, and Z 
are given and C is calculated. C. the correction, will 
result in the process waiting to perform the next 
event, performing the event, or advancing within 
the presentation to "catch up". In the case of 
40 record, T. Z, and C are given and E is calculated. 
C, the correction, is 0 since the time of the actual 
event is the same as the time of the recorded 
media event. Since C = 0. tiie general equation 
becomes: 

45 

E - (T - Z) = 0 

whic^ is algebraically equal to: 

50 E = T - Z 

In the example which follows, zero-time, Z, is not 
used in the equation. Instead coded zero-time, z, is 
used. Since z = -Z. because coded zero-time is 
55 two's complement of zero-time used to negate the 
ctock wrap problem, the general equation be- 
comes: 
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E - (T + z) = C 

As with the calculation of zero-time, use of 
zero-time is most easily understood in the context 
of a specific example. For this purpose, reference 
will be made to the numerical quantities in Rg. 7, 
while noting that these quantities and the resulting 
calculations are presented for illustrative purposes 
only. As shown, process 723 obtains coded zero- 
time 702 and stores it for frequent access. In 
particular, zero-time information 702 is FDE627EA 
(hex), and the time of event #1 726 on event 
schedule 725 is 0:00:00.000. Based on this in- 
formation, precisely at zero-time 701 process 723 
determines when to play the event. Current time 
724 is 9:47:25.600. which is 35,245,600 millisec- 
onds or 0219CE20 (hex). To determine how long 
the process 723 should wait for event #1 727, the 
sum of coded zero-time 702 and current time 724 
is subtracted from the time of event #1 726, yield- 
ing a result of 000009F6 (hex), which is 2550 
milliseconds. (That is: 00000000 (hex) 
(FDE627EA (hex) + 0219CE20 (hex)) = 000009F6 
(hex).) The result is that process 723 must wait 
2550 nnilliseconds before executing the event, i.e. 
playing event #1 727. 

After event #1 727 has been performed by 
process 723, process 723 immediately begins pre- 
paring for event #2 728. A media process, such as 
process 723, should accomplish as much prepara- 
tory work as possible before determining the next- 
event time, the time of event #n + 1 . By doing this, 
a media process will have the shortest code path 
length between the time the process is done wait- 
ing and the time the media event is to occur. This 
decreases the opportunity for variability in when 
the media event actually occurs. After process 723 
has completed as much preparatory work as possi- 
ble without actually performing event #2 728, it 
determines when to play event #2 728. Based on a 
time of event #2 729 of 0:00:03.250 on event 
schedule 725. it is noted that event #2 is to begin 
at 3250 milliseconds into the multimedia presenta- 
tion, or 00000CB2 (hex). Current time 730 is now 
9:47:29.950, which is 35.249,950 milliseconds or 
0219DF1E (hex). To determine how long the pro- 
cess 723 should wait for event #2 728. the sum of 
coded zero-time 702 and current time 730 is sub- 
tracted from the time of event #2 729, yielding a 
result of 000005AA (hex), which is 1450 millisec- 
onds. (That is: 00000CB2 (hex) - (FDE627EA (hex) 
+ 0219DF1E (hex)) = 000005AA (hex).) Thus, 
process 723 must wait 1450 milliseconds before 
playing event #2 728. 

The above procedure is used to determine the 
timing for each media event. NA^ere the result of 
the equation is positive, process 723 must wait the 
number of milliseconds given by the result. Where 



the result of the equation equals zero, process 723 
plays the event immediately. Where the result of 
the equation is negative, process 723 must "catch 
up** by the absolute value of the result in millisec- 

5 onds. Each of media playback processes may de* 
termine its own method for regaining synchroniza- 
tion, as described above. 

in certain circumstances, a time lag may occur 
between the starting of a media event and the 

70 actual presentation of the event to the observer. 
This may be caused by latent processing, hard- 
ware speed, or other dependencies. If the lag time 
is predictable, a correction for the lag time, called 
the correction time, can be appended to the cal- 

75 culations described above. In particular, the sum of 
coded zero-time 702 and current time is subtracted 
from the time of event #n 731 ; then the predicted 
amount of lag time is subtracted from this quantity. 
For streams of data, zero-time 701 is periodi- 

20 cally checked, by the process responsible for that 
media, against the current position in the stream to 
determine if the stream should be consumed at the 
same rate, accelerated, or decelerated. In this way, 
slight discrepancies between the playing speed of 

25 various media can be corrected dynamically. 

Figure 8 shows a representation of the use of 
the present invention in synchronizing multimedia 
during media repositioning commands. Media re- 
positioning commands may include fast fonward, 

30 fast rewind, stop, instant replay, rewind, etc. 

The example given in figure 8 represents how 
a user might interact with a system which uses 
zero-time based synchronization and how the sys- 
tem would respond. In this example, the user ini- 

35 tially wants to see the multimedia presentation from 
the beginning. After watching and listening for a 
minute or two, the user would like to see what's 
ahead in the multimedia presentation. The user fast 
forwards twenty-five minutes into the presentation 

40 and plays from that point for a little over two 
minutes. Realizing that there is some important 
information earlier in the multimedia presentation, 
the user rewinds back to the twelve minute mark 
and plays to the end of the presentation. 

45 The components of figure 8 are: A command 

sequence 801 which the user passes into the sys- 
tem 827. The time at which the commands are 
given are at the discretion of the user. A coordina- 
tion process 802 which receives the command 

50 sequence 801 and issues coordination messages 
828 829 830 through interprocess communication 
803 804 805 to each of the media processes 806 
807 808. A timeline demarcated by the passing of 
time in the common clock 201 will be used to 

55 demonstrate the activity within the system. 

At time 9:47:00.00 on the common clock 201. 
the user issues command #1 816 "Play" which is 
translated into "Play from 0:00:00" 809. This com- 
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mand is passed to the coordination process 802 
which performs the actions of the starter process 
described in the -PLAYING MULTIMEDIA" section. 
It determines zero-time #1 821, creates coded 
zero-time as described in the "CREATING CODED 
ZERO-TIME" section, initiates media processes 
806 807 808, and transmits 828 829 830 coded 
zero-time and the offset into the presentation at 
which to begin. 0:00:00.000 through interprocess 
communication 803 804 805. The media processes 

806 807 808 perform the actions of the playback 
processes described in the "PLAYING MULTI- 
MEDIA" section. They perform process prepara- 
tion, potentially perform process wait, and perform 
media events as described in the "USING CODED 
ZERO-TIME" section. 

The user decides after watching/listening to 
one minute and thirty seconds 824 of the presenta- 
tion that it's time to advance into the presentation. 
First, the user issues command #2 817 "Stop" 810 
which is sent to the coordination process 802 and 
transmitted 828 829 830 through interprocess com- 
munication 803 804 805 to each of the media 
processes 806 807 808. Each media process 806 

807 808 stops presenting its media. Second, the 
user issues command #3 - "Fast Forward" 811. 
This command is not sent to the media processes 
806 807 808; it is handled completely within the 
coordination process 802. The coordination pro- 
cess 802 gives the user a feedback as to how far 
"Fast Forward" has advanced into the multimedia 
presentation. These two commands 810 811 could 
be combined into one at the user interface level, 
but would be kept as two commands at the oper- 
ational level. 

When the user is notified that the multimedia 
presentation has been advanced to 0:25:15, the 
user issues command #4 818 "Play" which is 
translated into "Play from 0:25:15" 812. This com- 
mand is passed to the coordination process 802 
which performs the actions of the starter process 
described in the "PLAYING MULTIMEDIA" section. 
It determines zero-time #2 822, creates coded 
zero-time as described in the "CREATING CODED 
ZERO-TIME" section, and transmits 828 829 830 
coded zero-time and the offset into the presenta- 
tion at which to begin, 0:25:15.000, through Inter- 
process communication 803 804 805. The media 
processes 806 807 808 perform the actions of the 
playback processes described in the "PLAYING 
MULTIMEDIA" section. They perform process 
preparation, potentially perform process wait, and 
perform media events as described in the "USING 
CODED ZERO-TIME" section. 

The user decides after watching/listening to 
about two and one-quarter minutes 825 of the 
presentation that ifs time to return to an earlier 
spot in the presentation. Rrst. the user issues 



command #5 819 "Stop" 813 which is sent to the 
coordination process 802 and transmitted 828 829 
830 through interprocess communication 803 804 

805 to each of the media processes 806 807 808. 
5 Each media process 806 807 808 stops presenting 

its media. Second, the user issues command #6 - 
"Fast Rewind" 814. This command is not sent to 
the media processes 806 807 808; it is handled 
completely within the coordination process 802. 

10 The coordination process 802 gives the user a 
feedback as to how far "Fast Rewind" has backed 
up in the multimedia presentation. These two com- 
mands 813 814 could be combined into one at the 
user interface level, but would be kept as two 

75 commands at the operational level. 

When the user is notified that the multimedia 
presentation has backed up to 0:12:30. the user 
issues command #7 820 "Play" which is translated 
into "Play from 0:12:30" 815. This command is 

20 passed to the coordination process 802 which per- 
forms the actions of the starter process described 
in the "PLAYING MULTIMEDIA" section. It deter- 
mines zero-time #3 823. creates coded zero-time 
as described in the "CREATING CODED ZERO- 

25 TIME" section, and transmits 828 829 830 coded 
zero-time and the offset into the presentation at 
which to begin. 0:12:30.000, through interprocess 
communication 803 804 805. The media processes 

806 807 808 perform the actions of the playback 
30 processes described in the "PLAYING MULTI- 
MEDIA" section. They perform process prepara- 
tion, potentially perfonm process wait, and perform 
media events as described in the "USING CODED 
ZERO-TIME" section. Since the user does not is- 

35 sue another command, each media process 806 

807 808 continues performing 826 until it has no 
more events to perform, at which time it behaves 
as if it were sent a "Stop" command. 

While the invention has been described and 

40 illustrated in terms of synchronization of multi- 
media, it is to be noted that the principles and 
concepts presented may be readily adapted for 
use in synchronizing any computer-based process 
or processes, both in terms of initiation and on- 

45 going execution. Thus, for example, in a multipro- 
gramming computer system, performance measur- 
ing processes or vectorized processes requiring 
simultaneous data availability may be synchronized 
using a single common clock and a starter process 

50 to determine and broadcast a zero-time value in 
the same fashion as descritDed above. 

Additionally, while the foregoing description 
has included detailed discussion of the mathemat- 
ics used to determine zero-time, coded zero-time, 

55 and numerous offset/adjustment values, it is to be 
noted that many alternative schemes could be de- 
vised for combining values, separating values, scal- 
ing values, or shifting values. Thus, for instance, 

10 
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zero-time could be replaced by 1000-time, which 
additionally could be combined with the media-start 
time and used in combined form to drive synchro- 
nization for a particular media process. Such modi- 
fications are considered to be of form rather than s 
substance, and hence within the spirit and scope of 
the present invention. 

Using the foregoing specification, the invention 
may be implemented by standard programming 
and/or engineering techniques. The resulting io 
program (s) may be stored on disk, diskettes, mem- 
ory cards. ROM or any other memory device. For 
execution, the program (s) may be copied into a 
system memory (RAM) associated with a computer 
processor. One skilled in the art of computer sci- 75 
ence will readily be able to combine the system 
and process created as described above with ap- 
propriate general purpose or special purpose com- 
puter hardware to create a computer system em- 
bodying the invention. 20 

Claims 

1. A method for synchronizing initiation of media 
operations in a multimedia recording and 25 
playback system, comprising the steps of: 

in a starter process, receiving the current 
time from a clock means, assigning the re- 
ceived current time as an initialization-time val- 
ue, determining a zero-time value by adding a 30 
process preparation time to the initialization- 
time value, broadcasting the zero-time value to 
at least one media process; and 

in the media process, receiving the zero- 
time value, upon the current time reaching the 35 
zero-time value, initiating the media operation. 

2. A method as recited in claim 1, wherein the 
clock means is a system real-time clock. 

40 

3w A method as recited in claim 1, wherein the 
media process initiates its media operation in- 
dependently of other media processes. 

4. A method as recited in claim 3, wherein media 45 
process receives exactly one zero-time value 
transmission, and wherein the zero-time value 
transmission constitutes the only synchroniza- 
tion transmission initiated from the starter pro- 
cess to the media process. so 

a A method comprising the steps of: in the me- 
dia process, upon the current time reaching 
the zero-time value, repeating for each media 
event the sub*steps of obtaining the current 55 
time from the clock means, determining an 
event offset based on the zero-time value and 
the current time, logging the event offset in 



association with the media event, recording the 
media event 

6. A method as recited in claim 5, wherein the 
step of repeating for each media event further 
comprises the sub-step of, after recording the 
media event, waiting for arrival of the next 
media event. 

7. A method as recited in claim 5, wherein a 
media event includes any of a single presenta- 
tion object and a set of presentation objects. 

8. A method as recited in claim 5, wherein the 
logging step logs the event offset into the data 
rate of the recorded data. 

9. A method for synchronizing media recording 
as recited in claim 5, wherein the event offset 
is determined by subtracting the zero-time val- 
ue from the current time. 

10. A method as recited in claim 5. wherein the 
media process determines the granularity and 

number of media events independently of the 
starter process and other media processes. 

11. A method as claimed in any of claims 1 to 4, 
comprising the steps of: in the media process, 
accessing a log of event offset times asso- 
ciated with a plurality of media events, for each 
logged event offset time, performing the sub- 
steps of readying the associated media event 
for playing, and when the event offset time 
arrives, playing the media event. 

12. A method as recited in claim 11. wherein the 
event offset time arrival is determined by com- 
paring the current time with the sum of the 
zero-time value and the logged event offset 
time. 

13- A method as recited in claim 11, wherein the 
log of event offset times is provided in the data 
rate of the recorded data. 

14. A method as recited in claim 11. wherein the 
log of event offset times is created by any of 
recording and authorship. 

15- A method as recited in claim 11, wherein the 
step of performmg sub-steps further comprises 
the sub-steps of determining a correction time 
based on the event offset time, the current 
time, and zero-time, for a non-zero correction 
time, performing a resynchronization action on 
the next media event. 
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16. A method as recited in claim 11. wherein the 
correction time is determined by subtracting 
the current time from the sum of zero-time and 
the event offset time. 

5 

17. A method as recited in claim 11. wherein for a 
negative correction time the resynchronization 
action includes any of compression, truncation, 
and deletion. 

18. A method as recited in claim 11, wherein for a 
positive correction time the resynchronization 
action includes any of waiting and elongation. 

19. A method as recited in claim 5 or claim 11, is 
wherein the step of determining a zero-time 
value further includes coding the zero-time val- 
ue by computing its two's complement. 

20- A method as recited in claim 11. wherein the 20 
media process terminates itself upon playing 
its final media event. 
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@ System and method for synchronization of multimedia streams. 



@ A computer-based multimedia presentation sys- 
tem is provided with a synchronization scheme for 
recording and playing independent media. The dis- 
(v> closed system and method allows media processes 
^ and single medium processes to achieve and main- 
_ tain synchronization with each other without process 
interdependence and without interprocess commu- 
^ nication. This capability is provided by assigning a 
^ common clock for all processes, requiring all partici- 
g> pating media processes to reference the common 
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clock, informing each process of a synchronization 
t>asepoint called a "zero-time", and then allowing 
each process to independently synchronize itself to 
the common clock. The common clock itself does 
not provide any stimulus to a media process; it is a 
passive component in the synchronization. The me- 
dia process is the active component, referring to the 
common clock as required to maintain synchroniza- 
tion for the particular media it is handling. 
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